{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EB674E91811442DA06F0C1FA699779A",
    "mdEditEnable": false
   },
   "source": [
    "## 目录\n",
    "\n",
    "* Pandas数据结构Series：基本概念及创建\n",
    "* Pandas数据结构Series：索引\n",
    "* Pandas数据结构Series：基本技巧\n",
    "* Pandas数据结构Dataframe：基本概念及创建\n",
    "* Pandas数据结构Dataframe：索引\n",
    "* Pandas数据结构Dataframe：基本技巧\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "75F6498534EE4C8C8D5D1C66FA43000B",
    "mdEditEnable": false
   },
   "source": [
    "## 1.Pandas数据结构Series：基本概念及创建\n",
    "\n",
    "* \"一维数组\" Serise\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "C0A4F0ED70504B4088767F98CEB8319C",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 加载 warnings\n",
    "import warnings\n",
    "# 忽略 warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "267C21EAB16E42DC8317E29A1A0E5062",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Series 数据结构\n",
    "# Series 是带有标签的一维数组，可以保存任何数据类型（整数，字符串，浮点数，Python对象等）,轴标签统称为索引\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd  \n",
    "# 导入numpy、pandas模块\n",
    "\n",
    "s = pd.Series(np.random.rand(5))\n",
    "print(s)\n",
    "print(type(s))\n",
    "# 查看数据、数据类型\n",
    "\n",
    "print(s.index,type(s.index))\n",
    "print(s.values,type(s.values))\n",
    "# .index查看series索引，类型为rangeindex\n",
    "# .values查看series值，类型是ndarray\n",
    "\n",
    "# 核心：series相比于ndarray，是一个自带索引index的数组 → 一维数组 + 对应索引\n",
    "# 所以当只看series的值的时候，就是一个ndarray\n",
    "# series和ndarray较相似，索引切片功能差别不大\n",
    "# series和dict相比，series更像一个有顺序的字典（dict本身不存在顺序），其索引原理与字典相似（一个用key，一个用index）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "B0CA015F060B40578B40043F79FF04FD",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Series 创建方法一：由字典创建，字典的key就是index，values就是values\n",
    "\n",
    "dic = {'a':1 ,'b':2 , 'c':3, '4':4, '5':5}\n",
    "s = pd.Series(dic)\n",
    "print(s)\n",
    "# 注意：key肯定是字符串，假如values类型不止一个会怎么样？ → dic = {'a':1 ,'b':'hello' , 'c':3, '4':4, '5':5}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "F92E8836FB6743D5A2471C9B2A7F63ED",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Series 创建方法二：由数组创建(一维数组)\n",
    "\n",
    "arr = np.random.randn(5)\n",
    "s = pd.Series(arr)\n",
    "print(arr)\n",
    "print(s)\n",
    "# 默认index是从0开始，步长为1的数字\n",
    "\n",
    "s = pd.Series(arr, index = ['a','b','c','d','e'],dtype = np.object_)\n",
    "print(s)\n",
    "# index参数：设置index，长度保持一致\n",
    "# dtype参数：设置数值类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "BD39209DB35D41FBA8C99A9AA2A1CECD",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Series 创建方法三：由标量创建\n",
    "\n",
    "s = pd.Series(10, index = range(4))\n",
    "print(s)\n",
    "# 如果data是标量值，则必须提供索引。该值会重复，来匹配索引的长度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "E49344EEEC1B47C09BA06BCCA8F82F3B",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Series 名称属性：name\n",
    "\n",
    "s1 = pd.Series(np.random.randn(5))\n",
    "print(s1)\n",
    "print('-----')\n",
    "\n",
    "s2 = pd.Series(np.random.randn(5),name = 'test')\n",
    "print(s2)\n",
    "print(s1.name, s2.name,type(s2.name))\n",
    "print('-----')\n",
    "# name为Series的一个参数，创建一个数组的名称\n",
    "# .name方法：输出数组的名称，输出格式为str，如果没用定义输出名称，输出为None\n",
    "\n",
    "s3 = s2.rename('hehehe')\n",
    "print(s3)\n",
    "print(s3.name, s2.name)\n",
    "# .rename()重命名一个数组的名称，并且新指向一个数组，原数组不变"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "8DD84E4DD5DC48B18C96C5C1F81DB983",
    "mdEditEnable": false
   },
   "source": [
    "## 2.Pandas数据结构Series：索引\n",
    "\n",
    "* 位置下标 / 标签索引 / 切片索引 / 布尔型索引\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "821F50D578E94AAC8602E0DFB2FC745E"
   },
   "outputs": [],
   "source": [
    "# 位置下标，类似序列\n",
    "\n",
    "s = pd.Series(np.random.rand(5))\n",
    "print(s)\n",
    "print(s[0],type(s[0]),s[0].dtype)\n",
    "print(float(s[0]),type(float(s[0])))\n",
    "#print(s[-1])\n",
    "# 位置下标从0开始\n",
    "# 输出结果为numpy.float格式，\n",
    "# 可以通过float()函数转换为python float格式\n",
    "# numpy.float与float占用字节不同\n",
    "# s[-1]结果如何？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "B478090088FC439DBB7DED785EC283CC"
   },
   "outputs": [],
   "source": [
    "# 标签索引\n",
    "\n",
    "s = pd.Series(np.random.rand(5), index = ['a','b','c','d','e'])\n",
    "print(s)\n",
    "print(s['a'],type(s['a']),s['a'].dtype)\n",
    "# 方法类似下标索引，用[]表示，内写上index，注意index是字符串\n",
    "\n",
    "sci = s[['a','b','e']]\n",
    "print(sci,type(sci))\n",
    "# 如果需要选择多个标签的值，用[[]]来表示（相当于[]中包含一个列表）\n",
    "# 多标签索引结果是新的数组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "78D66F6017CB418E881BC2C304A0FD55"
   },
   "outputs": [],
   "source": [
    "# 切片索引\n",
    "\n",
    "s1 = pd.Series(np.random.rand(5))\n",
    "s2 = pd.Series(np.random.rand(5), index = ['a','b','c','d','e'])\n",
    "print(s1[1:4],s1[4])\n",
    "print(s2['a':'c'],s2['c'])\n",
    "print(s2[0:3],s2[3])\n",
    "print('-----')\n",
    "# 注意：用index做切片是末端包含\n",
    "\n",
    "print(s2[:-1])\n",
    "print(s2[::2])\n",
    "# 下标索引做切片，和list写法一样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "8B591D9F0B904EE781D12AD313062C5E"
   },
   "outputs": [],
   "source": [
    "# 布尔型索引\n",
    "\n",
    "s = pd.Series(np.random.rand(3)*100)\n",
    "s[4] = None  # 添加一个空值\n",
    "print(s)\n",
    "bs1 = s > 50\n",
    "bs2 = s.isnull()\n",
    "bs3 = s.notnull()\n",
    "print(bs1, type(bs1), bs1.dtype)\n",
    "print(bs2, type(bs2), bs2.dtype)\n",
    "print(bs3, type(bs3), bs3.dtype)\n",
    "print('-----')\n",
    "# 数组做判断之后，返回的是一个由布尔值组成的新的数组\n",
    "# .isnull() / .notnull() 判断是否为空值 (None代表空值，NaN代表有问题的数值，两个都会识别为空值)\n",
    "\n",
    "print(s[s > 50])\n",
    "print(s[bs3])\n",
    "# 布尔型索引方法：用[判断条件]表示，其中判断条件可以是 一个语句，或者是 一个布尔型数组！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "3F515D473071414996E4185FF0EE3DB0",
    "mdEditEnable": false
   },
   "source": [
    "## 3.Pandas数据结构Series：基本技巧\n",
    "\n",
    "* 数据查看 / 重新索引 / 对齐 / 添加、修改、删除值\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "92BF8B3ECA5B480289980393918DFB11"
   },
   "outputs": [],
   "source": [
    "# 数据查看\n",
    "\n",
    "s = pd.Series(np.random.rand(50))\n",
    "print(s.head(10))\n",
    "print(s.tail())\n",
    "# .head()查看头部数据\n",
    "# .tail()查看尾部数据\n",
    "# 默认查看5条"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "C699AFE40137487581EA0FE06606481F"
   },
   "outputs": [],
   "source": [
    "# 重新索引reindex\n",
    "# .reindex将会根据索引重新排序，如果当前索引不存在，则引入缺失值\n",
    "\n",
    "s = pd.Series(np.random.rand(3), index = ['a','b','c'])\n",
    "print(s)\n",
    "s1 = s.reindex(['c','b','a','d'])\n",
    "print(s1)\n",
    "# .reindex()中也是写列表\n",
    "# 这里'd'索引不存在，所以值为NaN\n",
    "\n",
    "s2 = s.reindex(['c','b','a','d'], fill_value = 0)\n",
    "print(s2)\n",
    "# fill_value参数：填充缺失值的值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "CC77BA730E0941868774159ECFB7F699"
   },
   "outputs": [],
   "source": [
    "# Series对齐\n",
    "\n",
    "s1 = pd.Series(np.random.rand(3), index = ['Jack','Marry','Tom'])\n",
    "s2 = pd.Series(np.random.rand(3), index = ['Wang','Jack','Marry'])\n",
    "print(s1)\n",
    "print(s2)\n",
    "print(s1+s2)\n",
    "# Series 和 ndarray 之间的主要区别是，Series 上的操作会根据标签自动对齐\n",
    "# index顺序不会影响数值计算，以标签来计算\n",
    "# 空值和任何值计算结果扔为空值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "78553424B4AB495D930685A0EE7638D9"
   },
   "outputs": [],
   "source": [
    "# 删除：.drop\n",
    "\n",
    "s = pd.Series(np.random.rand(5), index = list('ngjur'))\n",
    "print(s)\n",
    "s1 = s.drop('n')\n",
    "s2 = s.drop(['g','j'])\n",
    "print(s1)\n",
    "print(s2)\n",
    "print(s)\n",
    "# drop 删除元素之后返回副本(inplace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "92707904D39F4747BCC59659EB819934"
   },
   "outputs": [],
   "source": [
    "# 添加\n",
    "\n",
    "s1 = pd.Series(np.random.rand(5))\n",
    "s2 = pd.Series(np.random.rand(5), index = list('ngjur'))\n",
    "print(s1)\n",
    "print(s2)\n",
    "s1[5] = 100\n",
    "s2['a'] = 100\n",
    "print(s1)\n",
    "print(s2)\n",
    "print('-----')\n",
    "# 直接通过下标索引/标签index添加值\n",
    "s3 = s1 + s2\n",
    "# s3 = s1.append(s2)\n",
    "print(s3)\n",
    "print(s1)\n",
    "# 通过.append方法，直接添加一个数组(新版本使用+号)\n",
    "# .append方法生成一个新的数组，不改变之前的数组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "E3EC4AAB02894CB38E5CE14160B7DF37"
   },
   "outputs": [],
   "source": [
    "# 修改\n",
    "\n",
    "s = pd.Series(np.random.rand(3), index = ['a','b','c'])\n",
    "print(s)\n",
    "s['a'] = 100\n",
    "s[['b','c']] = 200\n",
    "print(s)\n",
    "# 通过索引直接修改，类似序列"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "5D9D527E5481424087CD29D1FD1501E4",
    "mdEditEnable": false
   },
   "source": [
    "## 4.Pandas数据结构Dataframe：基本概念及创建\n",
    "\n",
    "* \"二维数组\"Dataframe：是一个表格型的数据结构，包含一组有序的列，其列的值类型可以是数值、字符串、布尔值等。\n",
    "\n",
    "* Dataframe中的数据以一个或多个二维块存放，不是列表、字典或一维数组结构。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "id": "CC71E0B0DBEA4A808F6EBB9EE4F60D92"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   name  age gender\n",
      "0  Jack   18      m\n",
      "1   Tom   19      m\n",
      "2  Mary   20      w\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex(start=0, stop=3, step=1) \n",
      "该数据类型为： <class 'pandas.core.indexes.range.RangeIndex'>\n",
      "Index(['name', 'age', 'gender'], dtype='object') \n",
      "该数据类型为： <class 'pandas.core.indexes.base.Index'>\n",
      "[['Jack' 18 'm']\n",
      " ['Tom' 19 'm']\n",
      " ['Mary' 20 'w']] \n",
      "该数据类型为： <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 数据结构\n",
    "# Dataframe是一个表格型的数据结构，“带有标签的二维数组”。\n",
    "# Dataframe带有index（行标签）和columns（列标签）\n",
    "\n",
    "data = {'name':['Jack','Tom','Mary'],\n",
    "        'age':[18,19,20],\n",
    "       'gender':['m','m','w']}\n",
    "frame = pd.DataFrame(data)\n",
    "print(frame)  \n",
    "print(type(frame))\n",
    "print(frame.index,'\\n该数据类型为：',type(frame.index))\n",
    "print(frame.columns,'\\n该数据类型为：',type(frame.columns))\n",
    "print(frame.values,'\\n该数据类型为：',type(frame.values))\n",
    "# 查看数据，数据类型为dataframe\n",
    "# .index查看行标签\n",
    "# .columns查看列标签\n",
    "# .values查看值，数据类型为ndarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "id": "8EA12A19653340A38D13C9B4966348E1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6, 7]}\n",
      "{'one': array([0.41110197, 0.88308059, 0.30284832]), 'two': array([0.45381842, 0.27842142, 0.75943945])}\n",
      "   a  b  c\n",
      "0  1  3  5\n",
      "1  2  4  6\n",
      "2  3  5  7\n",
      "        one       two\n",
      "0  0.411102  0.453818\n",
      "1  0.883081  0.278421\n",
      "2  0.302848  0.759439\n",
      "   b  c  a    d\n",
      "0  3  5  1  NaN\n",
      "1  4  6  2  NaN\n",
      "2  5  7  3  NaN\n",
      "   b  c\n",
      "0  3  5\n",
      "1  4  6\n",
      "2  5  7\n",
      "         one       two\n",
      "f1  0.411102  0.453818\n",
      "f2  0.883081  0.278421\n",
      "f3  0.302848  0.759439\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 创建方法一：由数组/list组成的字典\n",
    "# 创建方法:pandas.Dataframe()\n",
    "\n",
    "data1 = {'a':[1,2,3],\n",
    "        'b':[3,4,5],\n",
    "        'c':[5,6,7]}\n",
    "data2 = {'one':np.random.rand(3),\n",
    "        'two':np.random.rand(3)}   # 这里如果尝试  'two':np.random.rand(4) 会怎么样？\n",
    "print(data1)\n",
    "print(data2)\n",
    "df1 = pd.DataFrame(data1)\n",
    "df2 = pd.DataFrame(data2)\n",
    "print(df1)\n",
    "print(df2)\n",
    "# 由数组/list组成的字典 创建Dataframe，columns为字典key，index为默认数字标签\n",
    "# 字典的值的长度必须保持一致！\n",
    "\n",
    "df1 = pd.DataFrame(data1, columns = ['b','c','a','d'])\n",
    "print(df1)\n",
    "df1 = pd.DataFrame(data1, columns = ['b','c'])\n",
    "print(df1)\n",
    "# columns参数：可以重新指定列的顺序，格式为list，如果现有数据中没有该列（比如'd'），则产生NaN值\n",
    "# 如果columns重新指定时候，列的数量可以少于原数据\n",
    "\n",
    "df2 = pd.DataFrame(data2, index = ['f1','f2','f3'])  # 这里如果尝试  index = ['f1','f2','f3','f4'] 会怎么样？\n",
    "print(df2)\n",
    "# index参数：重新定义index，格式为list，长度必须保持一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "id": "EC90FACDC85C498E8BA906B04EC908A9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'one': 0    0.052482\n",
      "1    0.192346\n",
      "dtype: float64, 'two': 0    0.006395\n",
      "1    0.368693\n",
      "2    0.342930\n",
      "dtype: float64}\n",
      "{'one': a    0.257066\n",
      "b    0.862857\n",
      "dtype: float64, 'two': a    0.170337\n",
      "b    0.472226\n",
      "c    0.553205\n",
      "dtype: float64}\n",
      "        one       two\n",
      "0  0.052482  0.006395\n",
      "1  0.192346  0.368693\n",
      "2       NaN  0.342930\n",
      "        one       two\n",
      "a  0.257066  0.170337\n",
      "b  0.862857  0.472226\n",
      "c       NaN  0.553205\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 创建方法二：由Series组成的字典\n",
    "\n",
    "data1 = {'one':pd.Series(np.random.rand(2)),\n",
    "        'two':pd.Series(np.random.rand(3))}  # 没有设置index的Series\n",
    "data2 = {'one':pd.Series(np.random.rand(2), index = ['a','b']),\n",
    "        'two':pd.Series(np.random.rand(3),index = ['a','b','c'])}  # 设置了index的Series\n",
    "print(data1)\n",
    "print(data2)\n",
    "df1 = pd.DataFrame(data1)\n",
    "df2 = pd.DataFrame(data2)\n",
    "print(df1)\n",
    "print(df2)\n",
    "# 由Seris组成的字典 创建Dataframe，columns为字典key，index为Series的标签（如果Series没有指定标签，则是默认数字标签）\n",
    "# Series可以长度不一样，生成的Dataframe会出现NaN值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "id": "C94962EEFBF44300AAC57592B816BA96"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.55414249 0.52144975 0.60090154]\n",
      " [0.53905384 0.07362346 0.89402133]\n",
      " [0.77145262 0.1607966  0.85723236]]\n",
      "          0         1         2\n",
      "0  0.554142  0.521450  0.600902\n",
      "1  0.539054  0.073623  0.894021\n",
      "2  0.771453  0.160797  0.857232\n",
      "        one       two     three\n",
      "a  0.554142  0.521450  0.600902\n",
      "b  0.539054  0.073623  0.894021\n",
      "c  0.771453  0.160797  0.857232\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 创建方法三：通过二维数组直接创建\n",
    "\n",
    "ar = np.random.rand(9).reshape(3,3)\n",
    "print(ar)\n",
    "df1 = pd.DataFrame(ar)\n",
    "df2 = pd.DataFrame(ar, index = ['a', 'b', 'c'], columns = ['one','two','three'])  # 可以尝试一下index或columns长度不等于已有数组的情况\n",
    "print(df1)\n",
    "print(df2)\n",
    "# 通过二维数组直接创建Dataframe，得到一样形状的结果数据，如果不指定index和columns，两者均返回默认数字格式\n",
    "# index和colunms指定长度与原数组保持一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "id": "6A990CE2CC42454B87EB9F069FA1739D"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'one': 1, 'two': 2}, {'one': 5, 'two': 10, 'three': 20}]\n",
      "   one  two  three\n",
      "0    1    2    NaN\n",
      "1    5   10   20.0\n",
      "   one  two  three\n",
      "a    1    2    NaN\n",
      "b    5   10   20.0\n",
      "   one  two\n",
      "0    1    2\n",
      "1    5   10\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 创建方法四：由字典组成的列表\n",
    "\n",
    "data = [{'one': 1, 'two': 2}, {'one': 5, 'two': 10, 'three': 20}]\n",
    "print(data)\n",
    "df1 = pd.DataFrame(data)\n",
    "df2 = pd.DataFrame(data, index = ['a','b'])\n",
    "df3 = pd.DataFrame(data, columns = ['one','two'])\n",
    "print(df1)\n",
    "print(df2)\n",
    "print(df3)\n",
    "# 由字典组成的列表创建Dataframe，columns为字典的key，index不做指定则为默认数组标签\n",
    "# colunms和index参数分别重新指定相应列及行标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "id": "DE8225EDDA4843CC84D9C7F0ABF6B077"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         Jack  Marry   Tom\n",
      "math       90     82  78.0\n",
      "english    89     95  67.0\n",
      "art        78     92   NaN\n",
      "         Jack   Tom  Bob\n",
      "math       90  78.0  NaN\n",
      "english    89  67.0  NaN\n",
      "art        78   NaN  NaN\n",
      "   Jack  Marry  Tom\n",
      "a   NaN    NaN  NaN\n",
      "b   NaN    NaN  NaN\n",
      "c   NaN    NaN  NaN\n"
     ]
    }
   ],
   "source": [
    "# Dataframe 创建方法五：由字典组成的字典\n",
    "\n",
    "data = {'Jack':{'math':90,'english':89,'art':78},\n",
    "       'Marry':{'math':82,'english':95,'art':92},\n",
    "       'Tom':{'math':78,'english':67}}\n",
    "df1 = pd.DataFrame(data)\n",
    "print(df1)\n",
    "# 由字典组成的字典创建Dataframe，columns为字典的key，index为子字典的key\n",
    "\n",
    "df2 = pd.DataFrame(data, columns = ['Jack','Tom','Bob'])\n",
    "df3 = pd.DataFrame(data, index = ['a','b','c'])\n",
    "print(df2)\n",
    "print(df3)\n",
    "# columns参数可以增加和减少现有列，如出现新的列，值为NaN\n",
    "# index在这里和之前不同，并不能改变原有index，如果指向新的标签，值为NaN （非常重要！）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "2ED109FF46ED405E88A16636D5FECF08",
    "mdEditEnable": false
   },
   "source": [
    "## 5.Pandas数据结构Dataframe：索引\n",
    "\n",
    "* Dataframe既有行索引也有列索引，可以被看做由Series组成的字典（共用一个索引）\n",
    "\n",
    "* 选择列 / 选择行 / 切片 / 布尔判断\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "id": "97E0B43DBCA848B186B9AC1E6A21350B"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               a          b          c          d\n",
      "one     3.652227  37.191870  33.768701  85.377864\n",
      "two    73.884595  32.906428  74.405845  45.000564\n",
      "three  75.847892   9.623630  99.297894  82.117513\n",
      "one       3.652227\n",
      "two      73.884595\n",
      "three    75.847892\n",
      "Name: a, dtype: float64 <class 'pandas.core.series.Series'>\n",
      "               a          c\n",
      "one     3.652227  33.768701\n",
      "two    73.884595  74.405845\n",
      "three  75.847892  99.297894 <class 'pandas.core.frame.DataFrame'>\n",
      "-----\n",
      "               a          c\n",
      "one     3.652227  33.768701\n",
      "two    73.884595  74.405845\n",
      "three  75.847892  99.297894 <class 'pandas.core.series.Series'>\n",
      "a     3.652227\n",
      "b    37.191870\n",
      "c    33.768701\n",
      "d    85.377864\n",
      "Name: one, dtype: float64 <class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "# 选择行与列\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(12).reshape(3,4)*100,\n",
    "                   index = ['one','two','three'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "\n",
    "data1 = df['a']\n",
    "data2 = df[['a','c']]\n",
    "print(data1,type(data1))\n",
    "print(data2,type(data2))\n",
    "print('-----')\n",
    "# 按照列名选择列，只选择一列输出Series，选择多列输出Dataframe\n",
    "\n",
    "data3 = df.loc['one']\n",
    "data4 = df.loc[['one','two']]\n",
    "print(data2,type(data3))\n",
    "print(data3,type(data4))\n",
    "# 按照index选择行，只选择一行输出Series，选择多行输出Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "3312914DAEB246B8BEF71DDF61BFB809"
   },
   "outputs": [],
   "source": [
    "# df[] - 选择列\n",
    "# 一般用于选择列，也可以选择行\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(12).reshape(3,4)*100,\n",
    "                   index = ['one','two','three'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "print('-----')\n",
    "\n",
    "data1 = df['a']\n",
    "data2 = df[['b','c']]  # 尝试输入 data2 = df[['b','c','e']]\n",
    "print(data1)\n",
    "print(data2)\n",
    "# df[]默认选择列，[]中写列名（所以一般数据colunms都会单独制定，不会用默认数字列名，以免和index冲突）\n",
    "# 单选列为Series，print结果为Series格式\n",
    "# 多选列为Dataframe，print结果为Dataframe格式\n",
    "\n",
    "data3 = df[:1]\n",
    "#data3 = df[0]\n",
    "#data3 = df['one']\n",
    "print(data3,type(data3))\n",
    "# df[]中为数字时，默认选择行，且只能进行切片的选择，不能单独选择（df[0]）\n",
    "# 输出结果为Dataframe，即便只选择一行\n",
    "# df[]不能通过索引标签名来选择行(df['one'])\n",
    "\n",
    "# 核心笔记：df[col]一般用于选择列，[]中写列名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "id": "769FE8B3FEC048C08EF99153BDD47012"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               a          b          c          d\n",
      "one    42.455249  45.603114  19.800797  19.275776\n",
      "two    96.801417  66.957034  75.543066  54.419541\n",
      "three  80.304868  77.408638  53.849315  63.658709\n",
      "four   38.459876  99.778942  85.430005  10.278021\n",
      "           a          b          c          d\n",
      "0  47.121220  87.290215  56.866733  38.311890\n",
      "1  26.166816  16.327043  33.535472  14.268213\n",
      "2  56.505928  34.759633  40.342344   6.929588\n",
      "3  12.434137  23.595751  81.762955  25.056899\n",
      "-----\n",
      "a    42.455249\n",
      "b    45.603114\n",
      "c    19.800797\n",
      "d    19.275776\n",
      "Name: one, dtype: float64\n",
      "a    26.166816\n",
      "b    16.327043\n",
      "c    33.535472\n",
      "d    14.268213\n",
      "Name: 1, dtype: float64\n",
      "单标签索引\n",
      "-----\n",
      "a     3.652227\n",
      "b    37.191870\n",
      "c    33.768701\n",
      "d    85.377864\n",
      "Name: one, dtype: float64\n",
      "           a          b          c          d\n",
      "3  12.434137  23.595751  81.762955  25.056899\n",
      "2  56.505928  34.759633  40.342344   6.929588\n",
      "1  26.166816  16.327043  33.535472  14.268213\n",
      "多标签索引\n",
      "-----\n",
      "               a          b          c          d\n",
      "one    42.455249  45.603114  19.800797  19.275776\n",
      "two    96.801417  66.957034  75.543066  54.419541\n",
      "three  80.304868  77.408638  53.849315  63.658709\n",
      "           a          b          c          d\n",
      "1  26.166816  16.327043  33.535472  14.268213\n",
      "2  56.505928  34.759633  40.342344   6.929588\n",
      "3  12.434137  23.595751  81.762955  25.056899\n",
      "切片索引\n"
     ]
    }
   ],
   "source": [
    "# df.loc[] - 按index选择行\n",
    "\n",
    "df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   index = ['one','two','three','four'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "df2 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df1)\n",
    "print(df2)\n",
    "print('-----')\n",
    "\n",
    "data1 = df1.loc['one']\n",
    "data2 = df2.loc[1]\n",
    "print(data1)\n",
    "print(data2)\n",
    "print('单标签索引\\n-----')\n",
    "# 单个标签索引，返回Series\n",
    "\n",
    "# data3 = df1.loc[['two','three','five']]\n",
    "data4 = df2.loc[[3,2,1]]\n",
    "print(data3)\n",
    "print(data4)\n",
    "print('多标签索引\\n-----')\n",
    "# 多个标签索引，如果标签不存在，则返回NaN（高版本则报错）\n",
    "# 顺序可变\n",
    "\n",
    "data5 = df1.loc['one':'three']\n",
    "data6 = df2.loc[1:3]\n",
    "print(data5)\n",
    "print(data6)\n",
    "print('切片索引')\n",
    "# 可以做切片对象\n",
    "# 末端包含\n",
    "\n",
    "# 核心笔记：df.loc[label]主要针对index选择行，同时支持指定index，及默认数字index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "0885E6B52C5C4859BDD46F8AC79979D3"
   },
   "outputs": [],
   "source": [
    "# df.iloc[] - 按照整数位置（从轴的0到length-1）选择行\n",
    "# 类似list的索引，其顺序就是dataframe的整数位置，从0开始计\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   index = ['one','two','three','four'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "print('------')\n",
    "\n",
    "print(df.iloc[0])\n",
    "print(df.iloc[-1])\n",
    "#print(df.iloc[4])\n",
    "print('单位置索引\\n-----')\n",
    "# 单位置索引\n",
    "# 和loc索引不同，不能索引超出数据行数的整数位置\n",
    "\n",
    "print(df.iloc[[0,2]])\n",
    "print(df.iloc[[3,2,1]])\n",
    "print('多位置索引\\n-----')\n",
    "# 多位置索引\n",
    "# 顺序可变\n",
    "\n",
    "print(df.iloc[1:3])\n",
    "print(df.iloc[::2])\n",
    "print('切片索引')\n",
    "# 切片索引\n",
    "# 末端不包含"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "E9711678A0E84C928257E7F2339D20EF"
   },
   "outputs": [],
   "source": [
    "# 布尔型索引\n",
    "# 和Series原理相同\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   index = ['one','two','three','four'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "print('------')\n",
    "\n",
    "b1 = df < 20\n",
    "print(b1,type(b1))\n",
    "print(df[b1])  # 也可以书写为 df[df < 20]\n",
    "print('------')\n",
    "# 不做索引则会对数据每个值进行判断\n",
    "# 索引结果保留 所有数据：True返回原数据，False返回值为NaN\n",
    "\n",
    "b2 = df['a'] > 50\n",
    "print(b2,type(b2))\n",
    "print(df[b2])  # 也可以书写为 df[df['a'] > 50]\n",
    "print('------')\n",
    "# 单列做判断\n",
    "# 索引结果保留 单列判断为True的行数据，包括其他列\n",
    "\n",
    "b3 = df[['a','b']] > 50\n",
    "print(b3,type(b3))\n",
    "print(df[b3])  # 也可以书写为 df[df[['a','b']] > 50]\n",
    "print('------')\n",
    "# 多列做判断\n",
    "# 索引结果保留 所有数据：True返回原数据，False返回值为NaN\n",
    "\n",
    "b4 = df.loc[['one','three']] < 50\n",
    "print(b4,type(b4))\n",
    "print(df[b4])  # 也可以书写为 df[df.loc[['one','three']] < 50]\n",
    "print('------')\n",
    "# 多行做判断\n",
    "# 索引结果保留 所有数据：True返回原数据，False返回值为NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "49A46D74170441FD8C27342B1EACDBFA"
   },
   "outputs": [],
   "source": [
    "# 多重索引：比如同时索引行和列\n",
    "# 先选择列再选择行 —— 相当于对于一个数据，先筛选字段，再选择数据量\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   index = ['one','two','three','four'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "print('------')\n",
    "\n",
    "print(df['a'].loc[['one','three']])   # 选择a列的one，three行\n",
    "print(df[['b','c','d']].iloc[::2])   # 选择b，c，d列的one，three行\n",
    "print(df[df['a'] < 50].iloc[:2])   # 选择满足判断索引的前两行数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "44F922FB6DC644ED8A5CED24570F7804",
    "mdEditEnable": false
   },
   "source": [
    "## 6.Pandas数据结构Dataframe：基本技巧\n",
    "\n",
    "* 数据查看、转置 / 添加、修改、删除值 / 对齐 / 排序\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "6A895647E14847578A013727566F5760"
   },
   "outputs": [],
   "source": [
    "# 数据查看、转置\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(8,2)*100,\n",
    "                   columns = ['a','b'])\n",
    "print(df.head(2))\n",
    "print(df.tail())\n",
    "# .head()查看头部数据\n",
    "# .tail()查看尾部数据\n",
    "# 默认查看5条\n",
    "\n",
    "print(df.T)\n",
    "# .T 转置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "0A9DE45F5245469C84ED124577E8DF28"
   },
   "outputs": [],
   "source": [
    "# 添加与修改\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "\n",
    "df['e'] = 10\n",
    "df.loc[4] = 20\n",
    "print(df)\n",
    "# 新增列/行并赋值\n",
    "\n",
    "df['e'] = 20\n",
    "df[['a','c']] = 100\n",
    "print(df)\n",
    "# 索引后直接修改值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "9E5AE74E01804BA78DF70AB53B0D18C1"
   },
   "outputs": [],
   "source": [
    "# 删除  del / drop()\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df)\n",
    "\n",
    "del df['a']\n",
    "print(df)\n",
    "print('-----')\n",
    "# del语句 - 删除列\n",
    "\n",
    "print(df.drop(0))\n",
    "print(df.drop([1,2]))\n",
    "print(df)\n",
    "print('-----')\n",
    "# drop()删除行，inplace=False → 删除后生成新的数据，不改变原数据\n",
    "\n",
    "print(df.drop(['d'], axis = 1))\n",
    "print(df)\n",
    "# drop()删除列，需要加上axis = 1，inplace=False → 删除后生成新的数据，不改变原数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "9757C543655B45048A4BBDDF8171133A"
   },
   "outputs": [],
   "source": [
    "# 对齐\n",
    "\n",
    "df1 = pd.DataFrame(np.random.randn(10, 4), columns=['A', 'B', 'C', 'D'])\n",
    "df2 = pd.DataFrame(np.random.randn(7, 3), columns=['A', 'B', 'C'])\n",
    "print(df1 + df2)\n",
    "# DataFrame对象之间的数据自动按照列和索引（行标签）对齐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "3F1B0FD39E5D44EF8AA4B2DF19C41C69"
   },
   "outputs": [],
   "source": [
    "# 排序1 - 按值排序 .sort_values\n",
    "# 同样适用于Series\n",
    "\n",
    "df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df1)\n",
    "print(df1.sort_values(['a'], ascending = True))  # 升序\n",
    "print(df1.sort_values(['a'], ascending = False))  # 降序\n",
    "print('------')\n",
    "# ascending参数：设置升序降序，默认升序\n",
    "# 单列排序\n",
    "\n",
    "df2 = pd.DataFrame({'a':[1,1,1,1,2,2,2,2],\n",
    "                  'b':list(range(8)),\n",
    "                  'c':list(range(8,0,-1))})\n",
    "print(df2)\n",
    "print(df2.sort_values(['a','c']))\n",
    "# 多列排序，按列顺序排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "id": "30F63C7B58E84CEE85CD6C6C17183B8C"
   },
   "outputs": [],
   "source": [
    "# 排序2 - 索引排序 .sort_index\n",
    "\n",
    "df1 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                  index = [5,4,3,2],\n",
    "                   columns = ['a','b','c','d'])\n",
    "df2 = pd.DataFrame(np.random.rand(16).reshape(4,4)*100,\n",
    "                  index = ['h','s','x','g'],\n",
    "                   columns = ['a','b','c','d'])\n",
    "print(df1)\n",
    "print(df1.sort_index())\n",
    "print(df2)\n",
    "print(df2.sort_index())\n",
    "# 按照index排序\n",
    "# 默认 ascending=True, inplace=False"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python-data-analysis",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
